How do animals weigh conflicting information about reward sources over time? Comparing dynamic averaging models.

Optimal foraging theory suggests that animals make decisions which maximize their food intake per unit time when foraging, but the mechanisms animals use to track the value of behavioral alternatives and choose between them remain unclear. Several models for how animals integrate past experience have been suggested. However, these models make differential predictions for the occurrence of spontaneous recovery of choice: a behavioral phenomenon in which a hiatus from the experimental environment results in animals reverting to a behavioral allocation consistent with a reward distribution from the more distant past, rather than one consistent with their most recently experienced distribution. To explore this phenomenon and compare these models, three free-operant experiments with rats were conducted using a serial reversal design. In Phase 1, two responses (A and B) were baited with pellets on concurrent variable interval schedules, favoring option A. In Phase 2, lever baiting was reversed to favor option B. Rats then entered a delay period, where they were maintained at weight in their home cages and no experimental sessions took place. Following this delay, preference was assessed using initial responding in test sessions where levers were presented, but not baited. Models were compared in performance, including an exponentially weighted moving average, the Temporal Weighting Rule, and variants of these models. While the data provided strong evidence of spontaneous recovery of choice, the form and extent of recovery was inconsistent with the models under investigation. Potential interpretations are discussed in relation to both the decision rule and valuation functions employed.

Resurgence Following Higher or Lower Quality Alternative Reinforcement.

Resurgence is a temporary increase in a previously suppressed target behavior following a worsening in reinforcement conditions. Previous studies have examined how higher rates or magnitudes of alternative reinforcement affect suppression of the target behavior and subsequent resurgence. However, there has been no investigation of the effects of higher versus lower qualities of alternative reinforcement on resurgence. Using a three-phase resurgence preparation with rats, the present experiments examined the effects of an alternative reinforcer that was of higher (Experiment 1) or lower (Experiment 2) quality than the reinforcer that had previously maintained the target behavior. The results of both experiments showed greater reductions in target behavior with a higher quality alternative reinforcer and larger increases in target responding when a higher quality alternative reinforcer was removed. Along with prior findings with higher rates and magnitudes of alternative reinforcement, these findings suggest that variations in reinforcer dimensions that increase the efficacy of alternative reinforcement also tend to increase resurgence when alternative reinforcement is removed. The results are discussed in terms of the resurgence as choice in context model and in terms of potential clinical implications.

Ketamine produces no detectable long-term positive or negative effects on cognitive flexibility or reinforcement learning of male rats.

RATIONALE: Patients with major depressive disorder (MDD) often experience abnormalities in behavioral adaptation following environmental changes (i.e., cognitive flexibility) and tend to undervalue positive outcomes but overvalue negative outcomes. The probabilistic reversal learning task (PRL) is used to study these deficits across species and to explore drugs that may have therapeutic value. Selective serotonin-reuptake inhibitors (SSRIs) have limited effectiveness in treating MDD and produce inconsistent effects in non-human versions of the PRL. As such, ketamine, a novel and potentially rapid-acting therapeutic, has begun to be examined using the PRL. Two previous studies examining the effects of ketamine in the PRL have shown conflicting results and only examined short-term effects of ketamine. OBJECTIVE: This experiment examined PRL performance across a 2-week period following a single exposure to a ketamine dose that varied across groups. METHODS: After five sessions of PRL training, groups of rats received an injection of either 0, 10, 20 or 30 mg/kg ketamine. One-hour post-injection, rats engaged in the PRL, and subsequently sessions continued daily for 2 weeks. Traditional behavioral and computational reinforcement learning-derived measures were examined. RESULTS: Results showed that ketamine had acute effects 1-h post-injection, including a significant decrease in the value of the punishment learning rate. Beyond 1 h, ketamine produced no detectable improvements nor decrements in performance across 2 weeks. CONCLUSION: Overall, the present results suggest that the range of ketamine doses examined do not have long-term positive or negative effects on cognitive flexibility or reward processing in healthy rats as measured by the PRL.

Examining resurgence in rats following expanded-operant treatments.

Resurgence of previously reinforced behavior represents a challenge to otherwise successful interventions based on differential reinforcement of alternative behavior (DRA). Expanded-operant treatments seek to increase the number of functional alternative behaviors through DRA, thereby potentially mitigating resurgence. However, the few studies that have directly examined these methods as a tool for resurgence mitigation have provided limited and unclear results. Thus, the present experiments were designed to investigate the effect of expanded-operant DRA methods on resurgence of previously reinforced behavior using rat subjects. In two experiments, following a baseline phase in which a target response was trained, groups of rats experienced concurrent (i.e., five simultaneous alternative responses), serial (i.e., five sequentially available alternative responses), or single DRA interventions arranging similar rates of alternative reinforcement in order to examine potential differences in resurgence. Both experiments showed that neither serial nor concurrent DRA expanded-operant treatments reduced resurgence compared with single DRA regardless of whether stimuli associated with previously reinforced alternative responses were removed (Experiment 1) or remained present (Experiment 2) for the serial-DRA group. Further, a primacy effect in resurgence was obtained for the serial-DRA group in both experiments. Overall, these results suggest that expanded-operant treatments may not help to reduce resurgence.

Further evaluation of treatment duration on the resurgence of destructive behavior.

Translation of promising procedures for mitigating treatment relapse has received considerable attention recently from researchers across the basic-applied continuum. One procedure that has demonstrated mixed support involves increasing the duration of treatment as a strategy for blunting resurgence. In a recent translational study, Greer et al. (2020) failed to detect a mitigation effect of increased treatment duration on the resurgence of destructive behavior. However, design limitations may have been responsible. The present study corrected these limitations by (a) employing a sequential design to decrease the possibility of multiple-treatment interference, (b) evaluating more treatment durations, (c) arranging treatments of fixed durations, and (d) conducting treatments of more extreme duration in a different clinical sample. Despite these improvements in experimental rigor and the testing of more extreme boundary conditions, the present study also failed to detect a mitigation effect of increased treatment duration. Likely explanations are discussed.

Basic and applied research on extinction bursts.

Discontinuation of the contingency between a response and its reinforcer sometimes produces a temporary increase in the response before its rate decreases, a phenomenon called the extinction burst. Prior clinical and basic studies on the prevalence of the extinction burst provide highly disparate estimates. Existing theories on the extinction burst fail to account for the dynamic nature of this phenomenon, and the basic behavioral processes that control response bursting remain poorly understood. In this paper, we first review the basic and applied literature on the extinction burst. We then describe a recent refinement of the concatenated matching law called the temporally weighted matching law that appears to resolve the above-mentioned issues regarding the extinction burst. We present illustrative translational data based conceptually on the model. Finally, we discuss specific recommendations derived from the temporally weighted matching law regarding procedures clinicians could implement to potentially mitigate or prevent extinction bursts.

Aversive control versus stimulus control by punishment.

Estes (1944) reported that adding electric shock punishment to extinction hastened response suppression but that responding increased when shock was removed. This result contributed to a view that reinforcement and punishment are asymmetrical processes because punishment has only indirect and temporary suppressive effects. Azrin and Holz (1966) suggested the result might be interpreted instead as shock serving as a discriminative stimulus for the absence of reinforcement. Here, to further examine potential stimulus control by punishment in a similar preparation, two groups of rats initially responded for food plus punishment and a third group for food alone. Reinforcement was then removed for all groups for the remaining three phases. With P and N denoting punishment and no punishment, the four phases for the three groups were: P-P-N-N, P-N-P-N, and N-P-N-N. We found some evidence for stimulus control by shock deliveries for group N-P-N-N (as suggested by Azrin and Holz), but all other changes in responding appeared due to introduction or removal of the aversive properties of shock. Although punishment may indeed have temporary effects under many circumstances, we argue that the view that this implies asymmetrical reinforcement and punishment processes was based on the flawed assumption that reinforcement has direct strengthening effects.

A Theory of the Extinction Burst.

A preliminary theory of a temporary increase in the rate of an operant response with the transition to extinction (i.e., the extinction burst) is proposed. The theory assumes reinforcers are events permitting access to some valuable activity, and that such activity can compete for allocation with the target response under some conditions (e.g., very high reinforcement rates). With the transition to extinction, elimination of this competition for allocation can produce an increase in the the target response, but the increase is transient because the value of the target response decreases with exposure to extinction. The theory provides a way to understand why the extinction burst is not ubiquitous, seems more common following very small ratio schedules, occurs for a short period of time following the transition to extinction, and may be eliminated with the availability of alternative reinforcement. It appears to provide a reasonable starting point for a theory of the extinction burst that does not necessarily require inclusion of invigorating effects of frustration, and it is closely aligned with Resurgence as Choice theory. Additional research on factors modulating reinforcement-related activities and how they affect the extinction burst could help to further evaluate the theory.

Nondrug reinforcers contingent on alternative behavior or abstinence increase resistance to extinction and reinstatement of ethanol-maintained behavior.

The effects of delivering nondrug alternative reinforcement on resistance to extinction and reinstatement of rats' ethanol-maintained lever pressing were evaluated in two experiments. In both, rats self-administered ethanol by lever pressing in a two-component multiple schedule during baseline. In the Rich component, alternative food reinforcement was made available for performing an alternative response (Experiment 1) or according to a differential-reinforcement-of-other-behavior schedule for lever pressing (Experiment 2). In the Lean component, only ethanol was available. Moreover, the frequency of alternative reinforcement was manipulated across conditions in Experiment 1. Following baseline, lever pressing was extinguished in both components by suspending ethanol reinforcement, and alternative food reinforcers were discontinued. Finally, to test for reinstatement, ethanol reinforcers were delivered independently of lever pressing in both components. In both experiments, proportion-of-baseline response rates were higher during extinction and reinstatement testing in the Rich component than in the Lean component (although differentiation was not observed at the lowest frequency of alternative reinforcement in Experiment 1). Thus, alternative nondrug reinforcers increased resistance to extinction and reinstatement of rats' ethanol-maintained lever pressing, even when those reinforcers were delivered contingently on an alternative response or on abstinence from lever pressing.

Effects of repeated exposure to escalating versus constant punishment intensity on response allocation.

The present experiment investigated the effects of 1) repeated exposures to escalating punishment intensities and 2) repeated exposure to punishment after periods of vacation on response allocation between punished and unpunished responding in three groups of rats. The first group (intensity + vacation) experienced repeated exposures to escalating punishment intensities after a period of vacation (i.e., return to baseline) from punishment. The second group (intensity-only) experienced repeated exposures to escalating punishment intensities without vacation from punishment. The third group (vacation-only) experienced repeated exposures to a constant punishment intensity after a period of vacation from punishment. Results showed that superimposition of punishment on one of two concurrently available responses decreased allocation toward the punished response and increased allocation toward the unpunished response. Furthermore, greater changes in allocation were observed with the introduction of a moderate constant intensity than with the introduction of a low intensity that increased across sessions. Reexposure to punishment had different effects between the groups. Although there was evidence that high shock intensities can enhance the efficacy of lower intensities to shift allocation away from the punished response and toward the unpunished response, there was little evidence of changes in response allocation with reintroduction of punishment after a period of vacation.

The effects of large, small, and thinning magnitudes of alternative reinforcement on resurgence.

Resurgence refers to an increase of a previously reinforced target behavior following the worsening of conditions for a more recently reinforced alternative behavior. There is evidence to suggest that alternative reinforcers of greater magnitude are more effective at reducing target responding but may also result in more resurgence when removed. Similar effects have been observed with high rates of alternative reinforcement. However, in clinical settings, reinforcement rate thinning is used to reduce the likelihood of resurgence associated with higher rates of alternative reinforcement. Given the clinical importance of alternative reinforcer magnitude, it is necessary to evaluate how reinforcer magnitude thinning may impact resurgence as well. Following Phase 1 in which target responding was reinforced, rats earned either large (six pellets), small (one pellet), or thinned (reduced from six pellets to one across sessions) magnitude reinforcement for alternative responding during target-response extinction in Phase 2. Then, alternative responding was placed on extinction for all groups in Phase 3. Target responding was comparably elevated at the end of Phase 2 for groups Small and Thin compared to group Large. In Phase 3, resurgence was evident only in group Large but target responding remained relatively elevated in groups Small and Thin. These results provide additional evidence of the important interplay between conditions of alternative reinforcement and the persistence and resurgence of target responding.

Resurgence Following Traditional and Interdependent Differential Reinforcement of Alternative Behavior.

Clinicians frequently prescribe functional communication training (FCT) as a treatment for severe destructive behavior. Recent research has shown that FCT treatments are susceptible to treatment relapse in the form of resurgence of destructive behavior when individuals contact periods in which reinforcers are unavailable. Results of preliminary studies suggest that teaching multiple response alternatives can mitigate the resurgence of target behavior. The current evaluation serves as a preliminary study in which we used a laboratory arrangement to evaluate the effects of a novel approach to training multiple alternative responses on the resurgence of target behavior. Findings showed that multiple-response training did not decrease resurgence of target responding consistently; however, it increased the total amount of target and alternative responding observed during the resurgence phase and decreased the overall probability of the target response.

The extinction burst: Impact of reinforcement time and level of analysis on measured prevalence.

Despite widespread belief in the extinction burst as a common occurrence, relatively little empirical work has focused directly on the phenomenon. In order to provide additional data on the topic, we report reanalyses of published extinction-control groups from our laboratory following training with a variety of schedules and reinforcers. In addition, two prospective experiments were conducted in which rats responded for food on FR 5 or FR 1 schedules prior to a within-session transition to extinction. The results of these reanalyses and experiments suggest that the obtained prevalence of the extinction burst was considerably greater when response rates in the first minute of the transition to extinction were considered as compared to when session-wide response rates were considered. In addition, when reinforcement time was included in baseline response-rate calculations, the obtained prevalence of the extinction burst was higher than when reinforcement time was omitted. These findings highlight the importance of measurement and definitional issues in the obtained prevalence of the extinction burst. Further, a closer alignment of such issues across basic and applied research would be desirable in terms of the development of future theories describing the processes giving rise to the extinction burst.

Destructive behavior increases as a function of reductions in alternative reinforcement during schedule thinning: A retrospective quantitative analysis.

The behavioral processes determining the magnitude of the resurgence of destructive behavior during reinforcement schedule thinning have yet to be described, despite an uptick in prevalence research on the topic. As predicted by Resurgence as Choice theory, recent animal research has found that resurgence increases with the magnitude of a downshift in alternative reinforcement. Here we reanalyze the data from 2 recent prevalence studies to determine whether the size of the decrease in alternative reinforcement availability predicts the magnitude of resurgence in the clinic. Results from this retrospective analysis suggest that resurgence of destructive behavior increases significantly with decreases in the availability of alternative reinforcement. Implications for future research and translations of theoretical analyses to the clinic are discussed.

Resurgence of alcohol seeking following abstinence induced by punishment in male and female rats.

To better approximate the human condition, animal models of relapse to drug and alcohol seeking have increasingly employed negative consequences to generate abstinence. Here we report the first demonstration of relapse to punishment-suppressed alcohol seeking induced by loss of non-drug reward (i.e., resurgence). We also report the first examination of potential sex differences in any form of relapse to alcohol seeking following suppression by punishment. Male and female rats first pressed a lever for 20 % oral alcohol. Next, lever pressing for one group continued to produce alcohol, but also produced occasional footshock. For another group, lever pressing similarly produced alcohol and occasional footshock, and a nose-poke response produced alternative non-drug reward (i.e., food). Males showed similar suppression of alcohol seeking by punishment alone and punishment + alternative non-drug reward, whereas females showed less suppression by punishment alone. Finally, when alternative reinforcement and punishment were suspended, resurgence occurred for both sexes in the group that previously had access to non-drug reward. Exposure to and then removal of punishment alone did not produce relapse for males, but it did for females. These results suggest that loss of alternative non-drug reward can generate relapse to alcohol seeking following abstinence induced by negative consequences. Future research should further examine the role of potential sex differences in sensitivity to punishment and how such differences may contribute to relapse more broadly.

Examination of alternative-response discrimination training and resurgence in rats.

Resurgence is an increase in a previously reinforced behavior following a worsening of conditions for a more recently reinforced behavior. Discrimination training is incorporated into treatment for problem behavior to prevent treatment adherence failures that may result in resurgence. There is evidence that resurgence may be reduced when a stimulus that signals alternative-response extinction is present compared with absent; however, the generality of this effect is unknown given the limited testing conditions. The goal of the present experiments was to further examine the effects of such stimuli in a reverse-translational evaluation using rats. Target responding was reinforced in baseline and then placed on extinction in the following discrimination-training phase. An alternative response was differentially reinforced in a two-component multiple schedule where one stimulus (i.e., SD) signaled alternative-response reinforcement and the other (i.e., SΔ) signaled extinction. Experiment 1 assessed resurgence in both the SΔ and SD when alternative reinforcement was removed. Experiment 2 evaluated resurgence under conditions that better approximated those used in the clinic in which the alternative-response SΔ was present or absent. The SΔ failed to suppress target responding during resurgence testing in both experiments. These findings suggest that the conditions under which an alternative-response SΔ will successfully mitigate resurgence may be limited and require further research.

Resurgence and repeated within-session progressive-interval thinning of alternative reinforcement.

Resurgence of a previously suppressed target behavior is common when reinforcement for a more recently reinforced alternative behavior is thinned. To better characterize such resurgence, these experiments examined repeated within-session alternative reinforcement thinning using a progressive-interval (PI) schedule with rats. In Experiment 1, a transition from a high rate of alternative reinforcement to a within-session PI schedule generated robust resurgence, but subsequent complete removal of alternative reinforcement produced no additional resurgence. Experiment 2 replicated these findings and showed similar effects with a fixed-interval (FI) schedule arranging similarly reduced session-wide rates of alternative reinforcement. Thus, the lack of additional resurgence following repeated exposure to the PI schedule was likely due to the low overall obtained rate of alternative reinforcement provided by the PI schedule, rather than to exposure to within-session reinforcement thinning per se. In both experiments, target responding increased at some point in the session during schedule thinning and continued across the rest of the session. Rats exposed to a PI schedule showed resurgence later in the session and after more cumulative alternative reinforcers than those exposed to an FI schedule. The results suggest the potential importance of further exploring how timing and change-detection mechanisms might be involved in resurgence.

Punishment and its putative fallout: A reappraisal.

In his book Coercion and Its Fallout Murray Sidman argued against the use of punishment based on concerns about its shortcomings and side effects. Among his concerns were the temporary nature of response suppression produced by punishment, the dangers of conditioned punishment, increases in escape and avoidance responses, punishment-induced aggression, and the development of countercontrol. This paper revisits Sidman's arguments about these putative shortcomings and side effects by examining the available data. Although Sidman's concerns are reasonable and should be considered when using any form of behavioral control, there appears to be a lack of strong empirical support for the notion that these potential problems with punishment are necessarily ubiquitous, long-lasting, or specific to punishment. We describe the need for additional research on punishment in general, and especially on its putative shortcomings and side effects. We also suggest the need for more effective formal theories of punishment that provide a principled account of how, why, and when lasting effects of punishment and its potential side effects might be expected to occur or not. In addition to being necessary for a complete account of behavior, such data and theories might contribute to improved interventions for problems of human concern.

Resurgence and downshifts in alternative reinforcement rate.

Resurgence refers to an increase in a previously suppressed target behavior with a relative worsening of conditions for a more recently reinforced alternative behavior. This experiment examined the relation between resurgence and the magnitude of a reduction in the rate of reinforcement for the alternative behavior. Groups of both male and female rats initially pressed a target lever for food on a variable-interval (VI) 30-s schedule. In a second phase, responding to the target lever was extinguished for all groups and pressing an alternative lever was reinforced on a VI 10-s schedule. Next, the rate of reinforcement for alternative behavior was reduced differentially across groups by arranging extinction, VI 80-s, VI 40-s, VI 20-s, or continued VI 10-s reinforcement. Target responding increased as an exponential function of the magnitude of the reduction in alternative reinforcement rates. With the exception that males appeared to show higher rates of target responding in baseline and higher rates of alternative responding in other phases, the overall pattern of responding across phases was not meaningfully different between sexes. The pattern of both target and alternative response rates across sessions and phases was well described quantitatively by the Resurgence as Choice in Context model.

Delays to food-predictive stimuli do not affect suboptimal choice in rats.

A variety of animals sometimes engage in a form of maladaptive decision-making characterized by repeatedly choosing an option providing food-predictive stimuli even though they earn less food for doing so. The temporal information-theoretic model suggests that such suboptimal choice depends on competition between the bits of temporal information conveyed by food-predictive stimuli (which encourages suboptimal choice) and the rate of food delivery (which encourages optimal choice). The model assumes that competition between these two sources of control is based on the ratio of the delay to food (Df) and the delay to food-predictive stimuli (Ds) at the choice point (i.e., Df/Ds). Research with both rats and pigeons suggests that temporal information outcompetes the rate of food delivery, thereby generating suboptimal choice, when the delay to food (Df) is sufficiently long. Limited data with pigeons, and none with rats, suggests that the rate of food delivery outcompetes temporal information, thereby generating optimal choice, when the delay to food-predictive stimuli (Ds) is sufficiently long. The present experiment sought to clarify whether longer delays to food-predictive stimuli decrease suboptimal choice in rats. We found that while longer delays to food (Df) increased suboptimal choice in rats, longer delays to food-predictive stimuli (Ds) did not decrease suboptimal choice. These results suggest a potential difference between rats and pigeons in the manner in which food-predictive stimuli and food itself compete to control choice. In terms of the temporal information-theoretic model, competition between temporal information and the rate of food delivery in rats appears to be influenced only by the delay to food at the choice point. (PsycInfo Database Record (c) 2020 APA, all rights reserved).